fix(scout): stop retiring realtime models on a wrong-endpoint 404 - #15
Merged
Merged
Conversation
Regression from the auto-retirement in #14. It disabled gemini-2.5-flash-native-audio-preview-12-2025 and gemini-3.1-flash-live-preview, both of which were working perfectly. The scout probes Google with generateContent. Realtime models answer only over bidiGenerateContent, so Google replies: "models/gemini-3.1-flash-live-preview is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods." That 404 describes the ENDPOINT, not the model. isGoneMessage matched on '404'/'not found' and three cycles later both rows were switched off. The misclassification predates #14 — status was already recorded as 'deprecated' — but nothing acted on it until retirement shipped, which is what turned a cosmetic wrong label into disabled models. Fixed at both layers: - isGoneMessage now rejects "is not supported for", "supported methods" and "not supported by". A wrong-method 404 is never removal, whatever the provider. This generalises past Google. - selectSweepCandidateIds no longer feeds realtime/speech/embedding-only rows to a chat probe at all. A model is swept when it advertises chat or vision, or advertises no capability (which the router already treats as unrestricted). The probe should not render a verdict it is not equipped to make. Migration V23 re-enables what the bug retired, scoped to rows whose only capabilities are realtime/audio so it restores exactly the class the probe cannot judge and never resurrects a genuinely dead chat model. It also clears gone_streak so a stale counter cannot re-retire them. Verified: the exact production error strings are now negative cases, while a genuine Google removal ("no longer available to new users") still retires.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regression from #14, caught by the maintainer within hours of deploy. The auto-retirement disabled
gemini-2.5-flash-native-audio-preview-12-2025andgemini-3.1-flash-live-preview— both working perfectly.What went wrong
The scout probes Google with
generateContent. Realtime models answer only overbidiGenerateContent, so Google replies:That 404 describes the endpoint, not the model.
isGoneMessagematched on404/not found, and three cycles later both rows were switched off atgone_streak=3.The misclassification predates #14 —
model_availability.statuswas already recording these asdeprecated. Nothing acted on it until retirement shipped, which is exactly what turned a cosmetic wrong label into disabled models. Retirement didn't create the bug; it made an existing one consequential.Fixed at both layers
1.
isGoneMessagerejects wrong-method 404s —is not supported for,supported methods,not supported by. A wrong-endpoint 404 is never removal, whatever the provider, so this generalises past Google.2.
selectSweepCandidateIdsstops feeding non-chat models to a chat probe. A model is swept when it advertiseschatorvision, or advertises no capability at all (which the router already treats as unrestricted). Realtime, speech and embedding-only rows are excluded — the probe shouldn't render a verdict it isn't equipped to make.Layer 2 is the real fix; layer 1 is the net that catches the same shape from any other provider.
Migration V23
Re-enables what the bug retired, scoped to rows whose only capabilities are realtime/audio — so it restores exactly the class the probe cannot judge and never resurrects a genuinely dead chat model. It also clears
gone_streak, so a stale counter can't re-retire them before the new candidate query takes effect.Verification
tsccleanno longer available to new users) still retires — the guard is narrow, not a blanket 404 exemptionNote on the deployed state
The two models are currently disabled on the VPS. V23 restores them on the next restart.
🤖 Generated with Claude Code